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Abstract 

In this analysis of extant data from 3,011 schools implementing school-wide positive behavioral 
interventions and supports (SWPBIS) across multiple years, we assessed the predictive power of 
various school characteristics and speed of initial implementation on sustained fidelity of 
implementation of SWPBIS at 1,3, and 5 years. In addition, we partitioned variance in whether 
schools sustained SWPBIS at the school, district, and state levels. Results showed that the largest 
differences in fidelity were at the state level. In addition, school characteristics (particularly 
grade levels served) and speed of initial implementation were significant but small predictors of 
sustained implementation. Results are discussed in terms of effective supports at the state and 
district levels, as well as addressing types of schools at greater risk for abandonment, such as 
high schools. 

Keywords: implementation science, positive behavior support, school¬ 
wide positive behavioral interventions and supports, external support, fidelity of 


implementation. 
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Predictors of Sustained Implementation of School-wide Positive Behavioral 

Interventions and Supports 

Sustainability of school-based interventions remains an important yet elusive goal for 
practitioners and researchers alike. Empirical research, anecdotal evidence, and theories on 
systems change have documented that implementation of systems-level school-based practices 
with fidelity requires a considerable amount of resources (Adelman & Taylor, 2003; Elias, Zins, 
Graczyk, & Weissburg, 2003). However, schools across the country are spending these resources 
to adopt practices that are unlikely to sustain beyond a few years at most (Fixsen, Naoom, Blase, 
Friedman, & Wallace, 2005). Furthermore, few large-scale empirical studies have been 
conducted into the variables that predict sustainability of practices in schools (McIntosh et al., 
2013). Identifying malleable variables that enhance sustainability of evidence-based practices is 
a much needed area of research. 

Sustainability has been defined as the continued implementation of a practice with 
ongoing fidelity of implementation of the foundational components, after resources used to 
support initial training and implementation have been withdrawn (Han & Weiss, 2005). The 
significance of sustaining evidence-based practices, however, stretches far beyond the monetary 
and personnel supports often provided to schools during initial training. For a program to be 
sustained with high fidelity of implementation, schools, districts, and states must attend to 
changes that schools inevitably face, including administrative turnover, loss of staff members, or 
new district priorities (Turri et al., 2014). These barriers to sustainability pose a great challenge 
for many schools, yet sustained implementation is vital to maintaining high-quality and effective 
practices in the long term (Fixsen et al., 2005). 
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One example of a school-based practice that has become an increased focus of 
implementation and sustainability research is school-wide positive behavioral interventions and 
supports (SWPBIS; Sugai & Horner, 2009). SWPBIS aims to promote a positive school climate 
and prevent disruptive behavior problems through the application of practices and systems 
consistent with the three-tiered public health prevention framework. Empirical studies conducted 
over the last 20 years have found that implementation of SWPBIS has been associated with (a) 
increases in academic achievement, positive student-teacher interactions, and classroom 
instructional time, (b) decreases in office discipline referrals, suspensions, and expulsions, and 
(c) improvements in school social climate and students’ social and emotional competencies (e.g. 
Barrett, Bradshaw, & Lewis-Palmer, 2008; Bradshaw, Mitchell, & Leaf, 2010; Homer et al., 
2009; Nelson, 1996; Scott & Barrett, 2004). 

Factors Influencing Sustained Implementation of School Practices 

Recent studies have identified a number of critical components that may be important to 
sustain evidence-based practices, including school demographic characteristics (McIntosh, Kim, 
Mercer, Strickland-Cohen, & Homer, in press; Payne, 2009), external supports at the district and 
state levels (Bradshaw & Pas, 2011; George & Kincaid, 2008) and the speed of initial 
implementation (Buzhardt, Greenwood, Abbott, & Tapia, 2006). 

School demographic characteristics. Researchers have noted that school demographic 
characteristics, particularly those related to racial and socioeconomic diversity, school size, and 
grade levels served, may be potentially influential in sustainability (Forman et al., 2013; 
McIntosh et al., in press). However, the empirical research in school characteristics as predictors 
of sustained implementation is inconsistent. A recent study found that schools with greater 
needs, particularly those with higher rates of suspension and mobility, were more likely to 
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receive training and had increased likelihood of program adoption than schools with higher 
student academic achievement (Bradshaw & Pas, 2011). Similarly, Payne (2009) found that 
schools located in economically disadvantaged areas with larger populations of African 
American students were more likely to engage in program selection and training at the local 
level, to choose a standardized program, and to integrate the chosen program into their normal 
school practices. The author hypothesized that this finding may be attributed to the resources 
provided to these schools through federal programs, which often come with the requirement of 
implementing evidence-based programs. In another related study, McIntosh and colleagues (in 
press) examined the association of school demographic characteristics with sustainability of 
SWPBIS and found that although grade levels served and years implementing were both 
significant predictors of sustainability, other factors, such as the percentage of students receiving 
free or reduced lunch and the percentage of non-white students, were not related to sustainability. 
Results showed that school team actions were more strongly associated with sustainability than 
school demographic characteristics. 

In 2010, Payne and Eckert conducted a national study of school violence interventions. 
Contrary to the findings previously noted, the authors found that high enrollment, urban locale, 
and high poverty were significant predictors of low implementation. Similarly, Gottfredson, 
Jones, and Gore (2002) found that larger populations of students of color and with low SES were 
factors related to low implementation and sustainability of a school-based cognitive-behavioral 
intervention. Such differences in empirical findings suggest the need for further research to 
identify what specific demographic factors play a critical role in the sustainability of school- 
based supports, which might indicate the need to provide more resources in implementation or 
modify practices to improve contextual fit. 
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External support. Although implementation of school practices is often dependent on 
classroom teachers (Han & Weiss, 2005), it is useful to examine how practices can be supported 
at the school, district, or state level. A few recent studies have examined critical features related 
to sustaining SWPBIS at the school (Mathews, McIntosh, Frank, & May, 2014), district 
(Bradshaw & Pas, 2011; George & Kincaid, 2008), and state (Homer et al., 2014). 

Mathews and colleagues (2014) investigated the extent to which critical features from the 
PBIS Self-Assessment Survey (SAS; Sugai, Homer, & Todd, 2000), a measure of fidelity of 
SWPBIS implementation, predicted sustained fidelity of implementation 3 years later, as 
measured by the Schoolwide Benchmark of Quality (BOQ; Kincaid, Childs, & George, 2005). 
Results showed that fidelity of implementation of classroom SWPBIS systems was the strongest 
predictor of sustained implementation. In particular, access to coaching in classrooms was one of 
the strongest predictors. This finding provides additional evidence of the importance of 
providing external support to classroom teachers for sustainable implementation (Blase, Van 
Dyke, & Fixsen, 2013; Joyce & Showers, 2002). 

The need for district-level support for schools implementing SWPBIS has grown as more 
district administrators begin to scale SWPBIS implementation across their schools. In a study of 
school and district level factors related to sustained implementation of SWPBIS, McIntosh and 
colleagues (2013) found that district level variables, particularly coaching and training of school- 
based personnel, were significant predictors of sustained SWPBIS implementation. George and 
Kincaid (2008) found that without careful planning around the supports required to successfully 
create a district-level model for implementation, such efforts will likely fail, as district personnel 
are often unfamiliar with the available resources necessary to implement and sustain systems 
change efforts. The authors outlined specific activities and tools, consistent with several elements 
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of the School-wide Positive Behavior Support: Implementation Blueprint and Self-Assessment 
(Sugai et al., 2010), for helping districts build the systems to scale up and sustain SWPBIS. 
Notably, the authors highlighted the importance of establishing a district-level leadership team, 
coaching and training capacity, an evaluation plan for assessing implementation and outcomes, 
and a long-term plan for securing funding. 

In 2011, Bradshaw and Pas documented the process by which the state of Maryland 
scaled up SWPBIS implementation and examined factors at the school and district levels that 
were associated with training, adoption, and implementation of SWPBIS. Consistent with 
findings published by Rohrbach and colleagues (2005), the authors found that training and 
adoption were predicted by district-level variables, suggesting the importance of the district in 
bringing schools into an initiative. Surprisingly, district factors appeared to be less related to 
fidelity of implementation. School-level factors, such as the number of years since a school was 
trained in SWPBIS and the concentration of qualified teachers, were stronger predictors of 
fidelity. These findings suggest the importance of district support, but also varying influence on 
different stages of implementation. 

State-level support for implementation has also been examined. In 2014, Horner et al. 
surveyed state-level PBIS coordinators and policy makers from seven states with at least 500 
schools implementing SWPBIS (Colorado, Florida, Illinois, Maryland, Missouri, North Carolina, 
and Oregon) to identify variables perceived as critical to the successful and sustained scaling-up 
of school-wide behavior support at each stage of implementation. Survey results suggested that 
for all seven states, the successful initial implementation of SWPBIS in 100 to 200 schools was 
necessary to recruit funding needed for scaling-up. However, initial pilot demonstrations at the 
school-level were seen as more likely to be sustained and lead to larger-scale implementation 
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when combined with concurrent efforts to build both district and state capacity in the areas of 
training, coaching, evaluation, and technical expertise. 

Speed of implementation. Another potential factor influencing sustainability is the 
length of time it takes from initial adoption to full implementation. Buzhardt and colleagues 
(2006) used rate of implementation as a measure to help identify barriers to implementation of an 
academic intervention and found considerable variability in the number of weeks needed to 
achieve full implementation as well as the mean time needed to complete each task. The 
researchers found that speed of implementation was related to the level of communication with 
research support staff—schools with more communication reached full implementation more 
quickly. None of the schools that were rated as having either delayed communication or limited 
communication reached full implementation. Although these results hint at the importance of 
external supports, other school-level factors (e.g., buy-in, effective leadership) may be as 
influential in implementation speed, and subsequently sustainability. 

The Present Study 

The studies reviewed provide some information regarding critical features that may 
predict adoption and sustained implementation of school practices, but many questions remain. 
For instance, the current research base is mixed regarding the effects of school demographics on 
fidelity of implementation. Additionally, the relative influence of variables at the school, district, 
and state levels over time has yet to be examined within one study, limiting our understanding of 
the most effective avenues for supporting schools. Further research examining school-level 
variables (e.g., school demographic characteristics and speed of implementation), in combination 
with assessing contributions of district and state, on longitudinal fidelity of implementation is 
needed to gain a better understanding of how implementation unfolds over time. Furthermore, a 
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large sample may help clarify conflicting findings from smaller samples. Such an approach may 
also identify which levels of supports—be they school, district, or state—are most critical to 
sustaining effective school practices. 

The purpose of this study was to assess which variables predict sustained implementation 
of SWPBIS across 5 years of implementation. Specifically, we used a national extant dataset to 
examine the relative importance of school-level demographic characteristics and speed of 
implementation in Year 1, while accounting for nesting within districts and states, in longitudinal 
implementation of SWPBIS to address the following research questions: 

1. Are there more between-school, between-district, or between-state differences in 
implementation at Years 1,3, and 5? 

2. What school-level variables predict implementation at Years 3 and 5? 

Method 


Participants and Settings 

All elementary, middle, and high schools in the United States meeting the following 
criteria were eligible for inclusion in the study: (1) at least 1 year of SWPBIS data reported to 
the OSEP National Technical Assistance Center on PBIS between the 2005-2006 and 2012-2013 
school years, (2) a Team Implementation Checklist (TIC; Sugai, Homer, & Lewis-Palmer, 2001) 
fidelity score reported during their first year of implementation, and (3) complete NCES school 
demographic data. A sequential cohort design was used, with the first year of SWPBIS fidelity 
data reported to the Center serving as the initial year of SWPBIS implementation and Year 3 and 
Year 5 defined as 3 and 5 years past the initial year of SWPBIS implementation. These years 
were selected because they roughly correspond to the stages of initial implementation, 
institutionalization, and ongoing evolution (sustainability) proposed by Adelman and Taylor 
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(1997). Each of these time periods has been theorized to present distinct challenges to schools, 
with implications regarding the relative need for and effectiveness of different types of external 
support (Mercer, McIntosh, Strickland-Cohen, & Homer, 2014). Because the main purpose of 
the study was to examine predictors of abandonment of SWPBIS, schools that stopped reporting 
fidelity data to the Center after Year 1 were included in the study. For the Year 3 analyses, 3,011 
schools were included because they had the potential for 3 years of SWPBIS implementation 
(i.e., initial SWPBIS implementation between 2005-2006 and 2010-2011). For the Year 5 
analyses, 1,242 schools were included with 5 years of potential SWPBIS implementation (i.e., 
initial implementation between 2005-2006 and 2007-2008). More detailed information on the 
schools in the Year 3 and Year 5 samples is presented in Table 1. 

Measures 

Fidelity of implementation. Multiple measures were used to assess SWPBIS fidelity of 
implementation. The School-wide Evaluation Tool (SET; Sugai et al., 2001) is a 28-item 
assessment that is completed by a trained external evaluator. The SET requires the school to 
meet or exceed a score of 80% implementation of SWPBIS critical features. Psychometric 
analyses reveal strong internal consistency (a = .96), test-retest reliability (mean = 97%) and 
inter-rater reliability (mean agreement = .99; Horner et al., 2004). The Schoolwide Benchmarks 
of Quality (BoQ; Kincaid et al., 2005) is an evaluation (either external or internal) that is based 
on observation and permanent product review. The criterion for adequate implementation of 
SWPBIS is 70%. Analyses of the measure’s psychometric properties indicate strong internal 
consistency (a = .96), test-retest reliability (r = .94), interrater reliability (r = .97), and concurrent 
validity (correlation with the SET, r = .51; Cohen, Kincaid, & Childs, 2007). The PBIS Self- 
Assessment Survey (SAS; Sugai et al., 2000) is a 43-item self-assessment measure with a 
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criterion for adequate implementation of 80%. Its correlation with the SET has been 
documented as .75 (Horner et al., 2004). Psychometric analysis shows strong internal 
consistency for subscale and total scores of the measure (a range = .85 to .94; Hagan-Burke et 
al., 2005; Safran, 2006). In this sample, concurrent validity between the measures at Year 1 was 
strong, with r = .63 (SET and BoQ), r = .56 (SET and SAS),, and r = .65 (SAS and BoQ), 
similar to previous research. 

Participating schools were identified as either sustaining or non-sustaining at Years 1, 3, 
and 5 using a dichotomous outcome variable (1 = fidelity measure indicating adequate 
implementation; 0 = no measures indicating adequate implementation). In cases in which schools 
had multiple measures during the year, the measure that was completed most externally to the 
school (and also had the highest reliability and internal consistency evidence) was used (i.e., 

SET, then BoQ, then SAS), in accordance with procedures established by McIntosh et al. (2013). 
The variable was calculated from the following measures (from the Year 5 subsample; Year 3 
were nearly identical): 31% SET, 14% BoQ, and 55% SAS. Additional evidence for using 
multiple SWPBIS measures to assess implementation comes from a recent study showing that 
these measures contribute to a latent SWPBIS Implementation variable with strong model fit 
(Turri et al., 2014). Composite internal consistency of this variable was .84, lending support for 
examining these measures within one variable. Schools that did not report any fidelity of 
implementation data to their district or the Center on PBIS were coded as not adequately 
implementing. 

The percent of schools coded by fidelity outcome were as follows. In Year 1, all schools 
reported a SWPBIS fidelity measure, and 79.6% of the Year 3 schools met or exceeded fidelity 
criteria in Year 1. In Year 3 in = 3,011), 60.9% of schools were at or above the fidelity criteria, 
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and 39.1% were not (8.6% were below the criteria, and 30.5% stopped reporting fidelity data). In 
Year 5 (n = 1,242), 38.8% of schools were at or above the fidelity criteria, and 61.2% were not 
(4.3% were below the criteria, and 56.8% had stopped reporting fidelity data). 

Speed of implementation. The Team Implementation Checklist (TIC; Sugai, Todd, & 
Homer, 2001) is a self-report measure used by school SWPBIS teams to monitor their progress 
in implementing key start-up and ongoing implementation activities. The participating schools 
completed two versions of the TIC (2.0 and 3.1), which have items that vary only slightly. Items 
are rated along a 3-point scale of implementation: achieved (i.e., all tasks completed), in progress 
(i.e., some tasks completed), or not started (i.e., none of the tasks completed). Team members are 
advised to complete the TIC quarterly until achieving adequate implementation (80%; Vincent & 
Tobin, 2012). Analyses of the measure’s psychometric properties indicate strong internal 
consistency (a = .91) and concurrent validity (correlation with the SET, r = .51; Cohen et al., 
2007). For this analysis, a combined TIC implementation average (% of total points) was used to 
assess the extent to which they made progress in completing initial SWPBIS implementation 
tasks within their first year of implementation. For schools that completed multiple TICs, the 
score from the last TIC administration of the year was used. 

Because prior psychometric investigations of the TIC did not treat item responses as 
ordinal, which is recommended when fewer than five response categories are used (Lubke & 
Muthen, 2004), ordinal confirmatory factor analyses (CFA) of the TIC 2.0 and 3.1 were 
conducted to ensure that the measure was adequate for the later analyses. Second-order CFA 
models were fit, with items loading on TIC subscales, and the TIC subscales loading on an 
overall TIC fidelity factor. Model fit for the TIC 3.1 was yj (202) = 1895.41, p < .001, CFI = .95, 
RMSEA = .06, with CFI and RMSEA values at Hu and Bender’s (1999) suggested cutoffs for 
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good fit. Model fit for the TIC 2.0 was similar, with X' (115) = 1550.19, p < .001, CFI = .95, 
RMSEA = .08; the CFI value remained at the Hu and Bentler cutoff, with the RMSEA slightly 
outside the cutoff value. Internal consistency of subscales and implementation average scores for 
both TIC versions was acceptable, with subscale ordinal alphas (Zumbo, Gadermann, & Zeisser, 
2007) ranging from .72 to .93 for the TIC 2.0 and .70 to .91 for the TIC 3.1, and ordinal alphas of 
.94 for the TIC 2.0 implementation average and .93 for the TIC 3.1 implementation average. 

School demographic characteristics. Schools reporting data to the Center on PBIS 
starting in the 2005-06 academic year were included in the study. Participating U.S. K-12 
schools totaled 3,011 in Year 3 and 1,242 in Year 5. School characteristic data were available 
from the National Center for Educational Statistics (NCES). In addition to district, grade level 
(elementary, middle, or high), and student enrollment, demographic data collected for each of the 
schools included: percentage of ethnic minority students, percentage of students receiving free 
and reduced lunch, and urbanicity (% of schools in: rural area, town, suburb, and city; per NCES 
criteria for population density). Data for schools in Years 3 and 5 are summarized in Table 1. 
Procedures 

All data for this study were extracted from an extant database maintained by the 
University of Oregon (2014). Schools entered SWPBIS fidelity data through a free online 
application (PBIS Assessment; pbisapps.org). Any school in the U.S. was eligible to report 
data—schools needed only to identify a local coordinator and agree to use of their data for 
research purposes to obtain a free account. 

Data Analysis 

To address research question 1 regarding the proportions of variance in SWPBIS fidelity 
at the school, district, and state levels, multilevel logistic regression models were fit with the 
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binary SWPBIS fidelity variable at Year 1, 3, or 5 as the criterion variable, with random 
intercepts at the district and state levels included to model between-district and -state differences 
in fidelity. To ensure that the same schools were compared across years, the Year 5 sample was 
used. Variance components from the model runs, with the school-level variance set to the 
variance of the logistic distribution (3.29; Snijders & Bosker, 2012), were used to calculate the 
proportion of variance at each level. 

To address research question 2 concerning predictors of sustained SWPBIS fidelity, three 
3-level logistic regression models were fit with Year 3 fidelity and then Year 5 fidelity as 
outcome variables. The first model was an unconditional model with only random intercepts at 
the district and state levels. The second model included school characteristics (i.e., school 
enrollment, percent of non-white students, percent of students eligible for free and reduced price 
meals (FARM), two dummy variables representing differences between middle and elementary 
schools and high vs. elementary schools, and school urbanicity). Because collinearity between 
the percent of non-white students and FARM prevented model convergence, only percent FARM 
was retained in the models (model results were nearly identical with percent of non-white 
students instead of FARM included). The third model added TIC scores as a predictor. To assess 
any differences in results by TIC version, a TIC version variable and an interaction term between 
version and TIC implementation average also were included in the model. Neither were 
statistically significant, so both were dropped from the reported models. All analyses were 
conducted using the glmer function of the lme4 package (Bates, Maechler, & Bolker, 2013) in R 
2.15.3 (R Core Team, 2013). As an indicator of effect size, marginal R~, which captures variance 
explained by the fixed but not random effects (Nakagawa & Schielzeth, 2013), is reported for the 
model with school characteristics and the model with Year 1 TIC scores. As an additional 
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indicator of effect size, odds ratios were calculated for statistically significant predictors of 
fidelity in the Year 3 and 5 models. Odds ratios below 1 indicate a decreased likelihood of 
SWPBIS fidelity, whereas odds ratios above 1 indicate an increased likelihood. 

Results 

Variance in Fidelity 

The percent of variance at the school, district, and state levels in Years 1,3, and 5 are 
presented in Table 2 and Figure 1. Across all three years, the largest differences in fidelity were 
between states (61% of variance in Year 1 to 40% in Year 5, M = 48.18%), with smaller, similar 
percentages of variance at the district (M = 24.16%) and school (M = 27.65%) levels. 

School-level Predictors 

As seen in Table 3, school grade level was the strongest school characteristic predictor of 
Year 3 and Year 5 SWPBIS fidelity. Compared to elementary schools, middle and high schools 
were less likely to meet SWPBIS fidelity criteria in both Year 3 and 5. In addition, schools with 
greater percentages of students eligible for FARM were less likely to implement to fidelity 
criteria in Year 3. Schools with higher Year 1 TIC scores were more likely to implement to 
SWPBIS criterion in both Years 3 and 5. 

It should be noted, however, that the models with school characteristics and TIC scores 
explained only a small amount of variance in Year 3 and Year 5 fidelity (marginal R = .02 to 
.04). The odds of middle schools, relative to elementary schools, of meeting the Year 3 fidelity 
criteria were .57, and .60 at Year 5. For high schools, the odds of meeting the Year 3 and 5 
fidelity criteria were .15 and .17, respectively, compared to elementary schools. Compared to a 
school with no students eligible for FARMs, a school with all students eligible had odds of .60 of 
meeting the fidelity criteria in Year 3. Last, schools with a Year 1 TIC score above the TIC 
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fidelity criterion, compared to schools below the TIC criterion, were 1.45 times more likely to be 
above the Year 3 fidelity criteria and 1.60 times more likely to be above the Year 5 criteria. 

Characteristics of Effective State Systems 

Because there were no state-level variables in the quantitative analyses, the high levels of 
variability at the state level prompted us to complete a descriptive post hoc analysis of states with 
considerable numbers of schools that sustained SWPBIS. In the quantitative analyses, there were 
five states with over 10 schools in the dataset and over 50% of schools that sustained 
implementation at 5 years. We interviewed state coordinators and technical assistance providers 
for these states to identify common features of their state systems. Each of these states had 
established state SWPBIS initiatives with the following features during the time of the study: a) 
an existing state SWPBIS leadership team, b) centralized SWPBIS trainings, with state-level 
trainers and standardized training curricula, c) recognition systems to acknowledge schools with 
strong implementation and/or outcomes, and d) regular use of the School-wide Positive Behavior 
Support: Implementation Blueprint and Self-Assessment (Sugai et al., 2010) to assess and inform 
state systems for supporting SWPBIS implementation. In contrast, each of the six states with 0% 
sustaining schools had none of these four features during that time. 

Discussion 

In efforts to examine sustained implementation of school practices and identify where the 
most variability in sustained implementation occurred, this study examined longitudinal 
implementation of SWPBIS and school-level predictors for a large, national sample of schools. 
Across 1, 3, and 5 years of implementation, the most variance in fidelity was at the state level, 
with slowly increasing variability related to district across years. Results highlight the 
importance of state-level systems of support for sustaining SWPBIS at the school level. Among 
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school-level predictors, school grade level was the strongest predictor, with elementary schools 
having higher odds of implementing at criterion than both middle and high schools. In addition, 
schools with lower rates of students receiving free and reduced price meals were more likely to 
sustain at high criterion. Finally, schools that met adequate criterion for implementation in Year 
1 were more likely to sustain. However, all of these school-level variables explained little 
variance in sustained implementation. 

Varying Importance of Support by Level 

The variance observed at the state level clearly points to the potential influence of state 
systems on sustained implementation of SWPBIS. Although there were no state-level predictors 
in the analyses, the results indicate that states may play a significant role in initial and sustained 
implementation, more so than school or district, at least within the first 5 years of 
implementation. The increasing proportion of variance at the district level indicates that district 
systems may play less of a role in speed of initial implementation but an increasing role as 
schools move toward 5 years. However, even at 5 years, the state appears to be more influential 
in sustainability than either the district or the school itself. These results provide some empirical 
support for the survey findings of Horner et al. (2014), which showed that capacity at the state- 
level was perceived as critical to ensuring the sustained implementation of SWPBIS. The results 
are also somewhat surprising, given the emphasis of previous research on school-level factors 
such as administrator support and staff commitment. These findings seem to align with research 
indicating that district coaching and capacity building play a key role in sustainability after initial 
implementation (Mathews et al., 2014; McIntosh et al., 2013); however, they also indicate the 
need for more implementation science research at the district and state levels to assess their 
relative importance beyond 5 years of implementation. 
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The descriptive results, although clearly tentative, indicate the importance of strong state 
leadership and the features identified to support schools in their implementation. Whether all of 
these features are necessary or equally effective remains unknown, but the features identified by 
the Implementation Blueprint seem to have been effective for these schools and, in the absence 
of further research, seem to be defensible recommendations for organizing state-level support for 
implementing and sustaining school practices until further research can be conducted. 

School Characteristics 

Results indicated that specific types of schools may have varying likelihoods of 
sustainability, and therefore abandonment. For example, middle and high schools were at greater 
risk of low implementation or abandonment. One important caveat to consider, however, is that 
the suggested timeframe for implementation is longer for high schools. Bohanon and colleagues 
(2006) suggested that instead of the 3- to 5-year commitment required in elementary and middle 
schools, a period of 5 to 8 years may be needed in high schools. Given the timeframe of this 
study, it is possible that although some high schools may technically have been below criterion 
in Years 3 and 5, they may have been on a trajectory to be implementing to criterion after this 
period. Nevertheless, this extended period of implementation may also increase risk for 
abandonment before the practice is even fully implemented, because implementers have not seen 
effects in terms of improved student outcomes (Andreou, McIntosh, Ross, & Kahn, in press). In 
light of the recent research showing positive effects of SWPBIS in high schools (Flannery, 
Fenning, Kato, & McIntosh, 2014), the results from this study highlight the need to support high 
schools during this fragile period of sub-criterion initial implementation, as well as descriptive 
research documenting the typical latency between initial training and full implementation in high 


schools. 
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Previous findings have indicated that school-level characteristics such as urbanicity, 
poverty, and high enrollment of students from traditionally underrepresented groups can serve as 
significant barriers to implementation of school-wide behavior support initiatives (Gottfredson et 
al., 2002; Payne & Eckert 2010). However, although our results did highlight some statistically 
significant school-level predictors, such as grade levels and poverty (i.e., students receiving free 
and reduced price meals), these school characteristics explained very little variance in sustained 
implementation. These results are encouraging, as they indicate the relative lack of true structural 
barriers to implementation. They may also explain the mixed results regarding school 
characteristics observed in previous research (Bradshaw & Pas, 2011; McIntosh et al., in press). 
These school characteristics may play a detectable role in sustained implementation, but other 
variables (e.g., features of the practice, specific district or state support) may mitigate these risks. 
Nevertheless, it is wise for practitioners to consider providing additional training and technical 
assistance to schools in high-poverty communities (Lorhmann & Davis, 2014, May) and high 
schools (Flannery, Sugai, & Anderson, 2009). 

Early Success in Implementation 

Similar to school characteristics, the speed of implementation, measured by TIC scores at 
Year 1, was a significant but small predictor of sustained implementation. In other words, those 
schools that reached criterion in Year 1 were somewhat more likely to sustain SWPBIS. One 
possible explanation for this outcome is that teams in these schools may have put enough 
components in place to see a rapid change in student outcomes that put them at a small 
advantage in relation to other schools. However, it is important to note that many schools that did 
not meet the TIC criterion in Year 1 went on to sustain SWPBIS by Year 5. Further, other 
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variables not measured in this study (e.g., strong principal leadership, staff commitment) may 
have enhanced both speed of implementation and sustainability. 

Limitations and Future Directions 

The analysis of extant data introduces several limitations to the study. Limitations include 
the use of multiple measures (including self report) to assess implementation and the nature of 
the sustained implementation criterion variable itself. Measuring at 3 and 5 years raises the 
possibility of meeting criteria in some years but not others (e.g., meeting in Years 1, 2, and 4 but 
not 3). In addition, schools that did not complete or report fidelity data may have been 
implementing SWPBIS adequately but without sharing data. However, because completing and 
reporting fidelity data is a key activity in SWPBIS implementation, it seems unlikely that these 
schools were implementing to criterion. Also, the dataset we used did not include additional 
variables that have previously been shown to affect implementation. Finally, the post hoc 
analyses were purely descriptive and not quantitative. The results regarding variability, 
especially at the state level, indicate the need for future quantitative research in this area. Such 
research could identify the most effective supports at the district and state levels for sustaining 


effective school practices. 
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Table 1 


School Characteristics for the Year 3 and Year 5 Samples 


Characteristic 

Year 3 Sample 

Year 5 Sample 

Number of Schools 

3,011 

1,242 

Number of Districts 

1,139 

530 

Number of States 

35 

29 

Student Enrollment, M ( SD) 

531 (369) 

504 (326) 

% Ethnic Minority Students, M (SD) 

38 (33) 

37 (33) 

% FARMs, M (SD) 

52 (25) 

50 (25) 

Grade Levels 

% Elementary Schools 

69 

70 

% Middle Schools 

21 

21 

% High Schools 

11 

9 

Urbanicity 

% Schools in Rural Areas 

29 

28 

% Schools in Towns 

18 

20 

% Schools in Suburbs 

27 

28 

% Schools in Cities 

26 

24 

Fidelity of SWPBIS Implementation 

% meeting or exceeding criteria 

61% 

39% 

% below criteria or not reporting 

39% 

61% 


Note. FARMs = free and reduced price meals. 
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Table 2 

Percent of Variance in SWPBIS Fidelity at School, District, and State Levels 


Year 1 Year 3 Year 5 


Level 

Variance 

% Variance 

Variance 

% Variance 

Variance 

% Variance 

School" 

3.29 

18.30 

3.29 

38.80 

3.29 

25.86 

District 

3.81 

21.19 

1.45 

17.10 

4.35 

34.20 

State 

10.88 

60.51 

3.74 

44.10 

5.08 

39.94 

Total 

17.98 

100.00 

8.48 

100.00 

12.72 

100.00 


Note, n = 1,242 schools in 530 districts in 29 states. a Variance components for school were fixed 
at 3.29. 
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Table 3 


Multilevel Logistic Regression Analyses Predicting Fidelity at Years 3 and 5 




Model 1 

Model 2 

Model 3 

Criterion 

Parameter 

Estimate SE 

Estimate 

SE 

Estimate 

SE 

Year 3 

Fixed Effects 
Intercept 

-0.29 0.32 

0.28 

0.39 

-0.51 

0.43 


Enrollment 


0.11 

0.07 

0.12 

0.07 


% FARMs 


-0.73 

0.26 

-0.72 

0.26 


Middle School 


-0.52*** 

0.12 

-0.50*** 

0.12 


High School 


-1.82*** 

0.19 

-1.74*** 

0.19 


Urbanicity 


-0.03 

0.06 

-0.03 

0.06 


Year 1 TIC 




1.18*** 

0.27 


Random Effects 
District 

1.63 

1.81 


1.78 



State 

Model Fit 

3.10 

3.34 


3.34 



Marginal R 2 


.03 


.04 



AIC 

3502 

3406 


3390 


Year 5 

Fixed Effects 
Intercept 

-1.77*** 0.49 

* 

-1.47 

0.63 

-2.79*** 

0.75 


Enrollment 


0.25 

0.13 

0.25 

0.13 


% FARMs 


-0.93 

0.55 

-0.97 

0.56 


Middle School 


-0.58 

0.24 

& 

-0.55 

0.25 


High School 


-1.97*** 

0.42 

-1.85*** 

0.43 


Urbanicity 


0.00 

0.12 

0.03 

0.13 


Year 1 TIC 




1.77 

0.53 


Random Effects 
District 

4.35 

4.94 


5.21 



State 

Model Fit 

5.08 

5.34 


5.40 



Marginal R 2 


.02 


.03 



AIC 

1288 

1273 


1265 



Note, n = 3,011 schools in 1,139 districts in 35 states (Year 3 models) and 1,242 schools in 530 


districts in 29 states (Year 5 models). FARMs = free and reduced price meals. AIC = Akaike 


Information Criterion. 

*p < .05, **p < .01, ***p < .001 
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Figure 1. Percent of variance infidelity of implementation at each level (school, district, state) 
by year. 



Year 1 Year 3 Year 5 



